In hosted services environments for multiple computer users, testing, evaluating, and deploying innovations to computer program applications, including web-based applications, can be an extremely time-consuming process. After an improvement to an existing application has been conceived, the application must be re-programmed to incorporate the improvement into the existing application, tested for errors, evaluated to determine the effectiveness of the improvement, and finally deployed for use by users in the hosted environment. For example, the deployment of a new web-based application may be introduced to change the location of an advertisement on a web page as a way to determine if the new location will generate more “clicks” and thus a greater potential for advertising revenue, from visitors to a hosted website.
In an attempt to speed up the deployment of new applications in hosted services environments, Control/Treatment (“CT”) “experiments,” (also called A/B tests, controlled experiments, randomized experiments, and parallel flights) may be utilized in “live” environments. In conventional CT experiments, live users may be randomly assigned to one of two variants: (i) the Control, which is commonly the “existing” version of an application, and (ii) the Treatment, which is usually a new version of the application being evaluated. Metrics of interest, ranging from runtime performance to implicit & explicit user behaviors and survey data, are collected. Statistical tests are then conducted on the data collected to evaluate whether there is a statistically significant difference between the two application versions, thus enabling the acceptance or rejection of hypotheses about the value of the new version. Currently, however, CT experiments require that either the existing version of the application be redeployed to incorporate the experimental features to be evaluated (thus creating the new version) or that an additional dependent programming layer be added to the existing version implementing application programming interfaces (“APIs”) which delegates user application requests to either the Control or Treatment versions based on how a user has been assigned. One drawback associated with application redeployment is that additional time and resources must be expended to stop the execution of the existing application (e.g., taking down a website) in order to modify the program to include the experimental features. One drawback associated with the additional programming layer is that this layer is dependent on the Control and Treatment application versions. Thus, when the APIs associated with the applications change, the programming code associated with the additional programming layer must also be changed since the layer implements the APIs. Further, the additional programming layer adds an extra “hop” or step in the data path between the requesting user and the applications, thus increasing latency. Still further, although the additional programming layer may be removed when experiments are not running, data center personnel must perform configuration changes (such as Domain Name System updates) to avoid impacting user client applications. It is with respect to these considerations and others that the various embodiments of the present invention have been made